Latent Space Research Articles

Exploiting a variational auto-encoder to represent the evolution of sudden stratospheric warmings

Sudden stratospheric warmings (SSWs) are the most dramatic events in the wintertime stratosphere. Such extreme events are characterized by substantial disruption to the stratospheric polar vortex, which can be categorized into displacement and splitting types depending on the morphology of the disrupted vortex. Moreover, SSWs are usually followed by anomalous tropospheric circulation regimes that are important for subseasonal-to-seasonal prediction. Thus, monitoring the genesis and evolution of SSWs is crucial and deserves further advancement. Despite several analysis methods that have been used to study the evolution of SSWs, the ability of deep learning methods has not yet been explored, mainly due to the relative scarcity of observed events. To overcome the limited observational sample size, we use data from historical simulations of the Whole Atmosphere Community Climate Model version 6 to identify thousands of simulated SSWs, and use their spatial patterns to train the deep learning model. We utilize a convolutional neural network combined with a variational auto-encoder (VAE)—a generative deep learning model—to construct a phase diagram that characterizes the SSW evolution. This approach not only allows us to create a latent space that encapsulates the essential features of the vortex structure during SSWs, but also offers new insights into its spatiotemporal evolution mapping onto the phase diagram. The constructed phase diagram depicts a continuous transition of the vortex pattern during SSWs. Notably, it provides a new perspective for discussing the evolutionary paths of SSWs: the VAE gives a better-reconstructed vortex morphology and more clearly organized vortex regimes for both displacement-type and split-type events than those obtained from principal component analysis. Our results provide an innovative phase diagram to portray the evolution of SSWs, in which particularly the splitting SSWs are better characterized. Our findings support the future use of deep learning techniques to study the underlying dynamics of extreme stratospheric vortex phenomena, and to establish a benchmark to evaluate model performance in simulating SSWs.

Whole-heart electromechanical simulations using Latent Neural Ordinary Differential Equations

Cardiac digital twins provide a physics and physiology informed framework to deliver personalized medicine. However, high-fidelity multi-scale cardiac models remain a barrier to adoption due to their extensive computational costs. Artificial Intelligence-based methods can make the creation of fast and accurate whole-heart digital twins feasible. We use Latent Neural Ordinary Differential Equations (LNODEs) to learn the pressure-volume dynamics of a heart failure patient. Our surrogate model is trained from 400 simulations while accounting for 43 parameters describing cell-to-organ cardiac electromechanics and cardiovascular hemodynamics. LNODEs provide a compact representation of the 3D-0D model in a latent space by means of an Artificial Neural Network that retains only 3 hidden layers with 13 neurons per layer and allows for numerical simulations of cardiac function on a single processor. We employ LNODEs to perform global sensitivity analysis and parameter estimation with uncertainty quantification in 3 hours of computations, still on a single processor.

PadGAN: An End-to-End dMRI Data Augmentation Method for Macaque Brain

Currently, an increasing number of macaque brain MRI datasets are being made publicly accessible. Unlike human, publicly accessible macaque brain datasets suffer from data quality in diffusion magnetic resonance imaging (dMRI) data. Typically, dMRI data require a minimum ratio of 1:10 between low b-value (b < 10) volumes and high b-value (b > 300) volumes. However, the currently accessible macaque datasets do not meet this ratio. Due to site differences in macaque brain images, traditional human brain image-to-image translation models struggle to perform well on macaque brain images. Our work introduces a novel end-to-end primary-auxiliary dual generative adversarial network (PadGAN) for generating low b-value images. The auxiliary generator in the PadGAN is responsible for extracting the latent space features from peak information maps and transmitting them to the primary generator, enabling the primary generator to generate images with rich details. Experimental results demonstrate that PadGAN outperforms existing methods both qualitatively and quantitatively (mean SSIM increased by 0.1139). Diffusion probabilistic tractography using dMRI data augmented by our method yields superior results.

Graph learning for particle accelerator operations

Particle accelerators play a crucial role in scientific research, enabling the study of fundamental physics and materials science, as well as having important medical applications. This study proposes a novel graph learning approach to classify operational beamline configurations as good or bad. By considering the relationships among beamline elements, we transform data from components into a heterogeneous graph. We propose to learn from historical, unlabeled data via our self-supervised training strategy along with fine-tuning on a smaller, labeled dataset. Additionally, we extract a low-dimensional representation from each configuration that can be visualized in two dimensions. Leveraging our ability for classification, we map out regions of the low-dimensional latent space characterized by good and bad configurations, which in turn can provide valuable feedback to operators. This research demonstrates a paradigm shift in how complex, many-dimensional data from beamlines can be analyzed and leveraged for accelerator operations.

Dual-space disentangled-multimodal network (DDM-net) for glioma diagnosis and prognosis with incomplete pathology and genomic data

Objective. Effective fusion of histology slides and molecular profiles from genomic data has shown great potential in the diagnosis and prognosis of gliomas. However, it remains challenging to explicitly utilize the consistent-complementary information among different modalities and create comprehensive representations of patients. Additionally, existing researches mainly focus on complete multi-modality data and usually fail to construct robust models for incomplete samples. Approach. In this paper, we propose a dual-space disentangled-multimodal network (DDM-net) for glioma diagnosis and prognosis. DDM-net disentangles the latent features generated by two separate variational autoencoders (VAEs) into common and specific components through a dual-space disentangled approach, facilitating the construction of comprehensive representations of patients. More importantly, DDM-net imputes the unavailable modality in the latent feature space, making it robust to incomplete samples. Main results. We evaluated our approach on the TCGA-GBMLGG dataset for glioma grading and survival analysis tasks. Experimental results demonstrate that the proposed method achieves superior performance compared to state-of-the-art methods, with a competitive AUC of 0.952 and a C-index of 0.768. Significance. The proposed model may help the clinical understanding of gliomas and can serve as an effective fusion model with multimodal data. Additionally, it is capable of handling incomplete samples, making it less constrained by clinical limitations.

CTGGAN: Controllable Text Generation with Generative Adversarial Network

Controllable Text Generation (CTG) aims to modify the output of a Language Model (LM) to meet specific constraints. For example, in a customer service conversation, responses from the agent should ideally be soothing and address the user’s dissatisfaction or complaints. This imposes significant demands on controlling language model output. However, demerits exist among traditional methods. Promoting and fine-tuning language models exhibit the “hallucination” phenomenon and cannot guarantee complete adherence to constraints. Conditional language models (CLM), which map control codes into LM representations or latent space, require training the modified language models from scratch and a high amount of customized dataset is demanded. Decoding-time methods employ Bayesian Rules to modify the output of the LM or model constraints as a combination of energy functions and update the output along the low-energy direction. Both methods are confronted with the efficiency sampling problem. Moreover, there are no methods that consider the relation between constraints weights and the contexts, as is essential in actual applications such as customer service scenarios. To alleviate the problems mentioned above, we propose Controllable Text Generation with Generative Adversarial Networks (CTGGAN), which utilizes a language model with logits bias as the Generator to produce constrained text and employs the Discriminator with learnable constraint weight combinations to score and update the generation. We evaluate the method in the text completion task and Chinese customer service dialogues scenario, and our method shows superior performance in metrics such as PPL and Dist-3. In addition, CTGGAN also exhibits efficient decoding compared to other methods.

A Novel Obstacle Traversal Method for Multiple Robotic Fish Based on Cross-Modal Variational Autoencoders and Imitation Learning

Precision control of multiple robotic fish visual navigation in complex underwater environments has long been a challenging issue in the field of underwater robotics. To address this problem, this paper proposes a multi-robot fish obstacle traversal technique based on the combination of cross-modal variational autoencoder (CM-VAE) and imitation learning. Firstly, the overall framework of the robotic fish control system is introduced, where the first-person view of the robotic fish is encoded into a low-dimensional latent space using CM-VAE, and then different latent features in the space are mapped to the velocity commands of the robotic fish through imitation learning. Finally, to validate the effectiveness of the proposed method, experiments are conducted on linear, S-shaped, and circular gate frame trajectories with both single and multiple robotic fish. Analysis reveals that the visual navigation method proposed in this paper can stably traverse various types of gate frame trajectories. Compared to end-to-end learning and purely unsupervised image reconstruction, the proposed control strategy demonstrates superior performance, offering a new solution for the intelligent navigation of robotic fish in complex environments.

Evolution of laughter from play

ABSTRACT In this hypothesis, I discuss how laughter from physical play could have evolved to being induced via visual or even verbal stimuli, and serves as a signal to highlight incongruity that could potentially pose a threat to survival. I suggest how laughter’s induction could have negated the need for physical contact in play, evolving from its use in tickling, to tickle-misses, and to taunting, and I discuss how the application of deep learning neural networks trained on images of spectra of a variety of laughter types from a variety of individuals or even species, could be used to determine such evolutionary pathways via the use of latent space exploration.

Individualized dynamic model for multi-resolutional data

SUMMARY Mobile health has emerged as a major success for tracking individual health status, due to the popularity and power of smartphones and wearable devices. This has also brought great challenges in handling heterogeneous, multi-resolution data which arise ubiquitously in mobile health due to irregular multivariate measurements collected from individuals. In this paper, we propose an individualized dynamic latent factor model for irregular multi-resolution time series data to interpolate unsampled measurements of time series with low resolution. One major advantage of the proposed method is the capability to integrate multiple irregular time series and multiple subjects by mapping the multi-resolution data to the latent space. In addition, the proposed individualized dynamic latent factor model is applicable to capturing heterogeneous longitudinal information through individualized dynamic latent factors. Our theory provides a bound on the integrated interpolation error and the convergence rate for B-spline approximation methods. Both the simulation studies and the application to smartwatch data demonstrate the superior performance of the proposed method compared to existing methods.

Advanced Flame front Detection in Combustion Processes Using Autoencoder Approach

This research explores the detection of flame front evolution in spark-ignition engines using an innovative neural network, the autoencoder. High-speed camera images from an optical access engine were analyzed under different air excess coefficient λ conditions to evaluate the autoencoder’s performance. This study compared this new approach (AE) with an established method used by the same research group (BR) across multiple combustion cycles. Results revealed that the AE method outperformed the BR in accurately identifying flame pixels and significantly reducing overestimations outside the flame boundary. AE exhibited higher sensitivity levels, indicating its superior ability to identify pixels and minimize errors compared to the BR method. Additionally, AE’s accuracy in representing combustion evolution was notably improved, offering a more detailed depiction of the process. AE’s strength lies in its independence from specific threshold searches, a requirement in the BR method. By relying on learned representations within its latent space, AE eliminates laborious threshold exploration, ensuring reliability and reducing workload pressures. Comparative analyses consistently confirmed AE’s superior performance in accurately reproducing and delineating combustion evolution compared to BR. This study highlights AE’s potential as a promising technique for precise flame front detection in combustion processes. Its ability to autonomously extract features, minimize errors, and enhance overall accuracy signifies a significant step forward in analyzing flame fronts. AE’s reliability, reduced need for manual intervention, and adaptability across various conditions suggest a promising future for improving combustion analysis techniques in spark-ignition engines with optical access.

Examining Differential Item Functioning from a Multidimensional IRT Perspective.

Differential item functioning (DIF) is a standard analysis for every testing company. Research has demonstrated that DIF can result when test items measure different ability composites, and the groups being examined for DIF exhibit distinct underlying ability distributions on those composite abilities. In this article, we examine DIF from a two-dimensional multidimensional item response theory (MIRT) perspective. We begin by delving into the compensatory MIRT model, illustrating and how items and the composites they measure can be graphically represented. Additionally, we discuss how estimated item parameters can vary based on the underlying latent ability distributions of the examinees. Analytical research highlighting the consequences of ignoring dimensionally and applying unidimensional IRT models, where the two-dimensional latent space is mapped onto a unidimensional, is reviewed. Next, we investigate three different approaches to understanding DIF from a MIRT standpoint: 1. Analytically Uniform and Nonuniform DIF: When two groups of interest have different two-dimensional ability distributions, a unidimensional model is estimated. 2. Accounting for complete latent ability space: We emphasize the importance of considering the entire latent ability space when using DIF conditional approaches, which leads to the mitigation of DIF effects. 3. Scenario-Based DIF: Even when underlying two-dimensional distributions are identical for two groups, differing problem-solving approaches can still lead to DIF. Modern software programs facilitate routine DIF procedures for comparing response data from two identified groups of interest. The real challenge is to identify why DIF could occur with flagged items. Thus, as a closing challenge, we present four items (Appendix A) from a standardized test and invite readers to identify which group was favored by a DIF analysis.

Multi-Modal Latent Diffusion

Multimodal datasets are ubiquitous in modern applications, and multimodal Variational Autoencoders are a popular family of models that aim to learn a joint representation of different modalities. However, existing approaches suffer from a coherence–quality tradeoff in which models with good generation quality lack generative coherence across modalities and vice versa. In this paper, we discuss the limitations underlying the unsatisfactory performance of existing methods in order to motivate the need for a different approach. We propose a novel method that uses a set of independently trained and unimodal deterministic autoencoders. Individual latent variables are concatenated into a common latent space, which is then fed to a masked diffusion model to enable generative modeling. We introduce a new multi-time training method to learn the conditional score network for multimodal diffusion. Our methodology substantially outperforms competitors in both generation quality and coherence, as shown through an extensive experimental campaign.

The flattening of spacetime hierarchy of the DMT brain state is characterised by harmonic decomposition of spacetime (HADES) framework

Abstract The human brain is a complex system, whose activity exhibits flexible and continuous reorganisation across space and time. The decomposition of whole-brain recordings into harmonic modes has revealed a repertoire of gradient-like activity patterns associated with distinct brain functions. However, the way these activity patterns are expressed over time with their changes in various brain states remains unclear. Here, we investigate healthy participants taking the serotonergic psychedelic N, N-Dimethyltryptamine (DMT) with the Harmonic Decomposition of Spacetime (HADES) framework that can characterise how different harmonic modes defined in space are expressed over time. HADES demonstrates significant decreases in contributions across most low-frequency harmonic modes in the DMT-induced brain state. When normalizing the contributions by condition (DMT and non-DMT), we detect a decrease specifically in the second functional harmonic, which represents the uni- to transmodal functional hierarchy of the brain, supporting the leading hypothesis that functional hierarchy is changed in psychedelics. Moreover, HADES’ dynamic spacetime measures of fractional occupancy, life time and latent space provide a precise description of the significant changes of the spacetime hierarchical organization of brain activity in the psychedelic state.

Multi-step planning with learned effects of partial action executions

In this paper, we propose a novel affordance model, which combines object, action, and effect information in the latent space of a predictive neural network architecture that is built on Conditional Neural Processes. Our model allows us to make predictions of intermediate effects expected to be obtained during action executions and make multi-step plans that include partial actions. We first compared the prediction capability of our model using an existing interaction data set and showed that it outperforms a recurrent neural network-based model in predicting the effects of lever-up actions. Next, we showed that our model can generate accurate effect predictions for other actions, such as push and grasp actions. Our system was shown to generate successful multi-step plans to bring objects to desired positions using the traditional A* search algorithm. Furthermore, we realized a continuous planning method and showed that the proposed system generated more accurate and effective plans with sequences of partial action executions compared to plans that only consider full action executions using both planning algorithms.

Multi-DORGP for fast uncertainty quantification of multi-scale irregular defects in super large-scale fiber-reinforced composite

The presence of randomly distributed irregular defects in fiber-reinforced composites significantly impacts the structural response, yet traditional machine learning schemes often require an extensive number of samples, resulting in time-consuming processes. Therefore, this paper introduces a novel approach, termed the dual order-reduced Gaussian Process emulators (Multi-DORGP), aimed at efficiently quantifying the uncertainty of multi-scale irregular defects in fiber-reinforced composites. This is achieved by accurately characterizing the multi-scale irregular defects through precise geometric representations, utilizing massive discrete nodes and fine meshes to address the limitations of parameterization methods that may overlook shape and size differences of defects. Moreover, we comprehensively quantify uncertainty across micro, meso, and macro scales, considering spatially randomly distributed locations, sizes, and irregular shapes of defects. Notably, the Multi-DORGP scheme is presented to alleviate the computational burden associated with extremely high-dimensional data (e.g., up to 23.3 million variables). In which, we decouple and build the latent spaces for raw data assisted by principal component analysis, and Gaussian Process regression is built and trained by the coefficients between the latent spaces. Through illustrative examples, including a real-life application involving an airplane wing, we validate the proposed scheme's capability to accurately quantify the uncertainty of multi-scale irregular defects in large-scale fiber-reinforced composites using significantly few samples (e.g., 100).

StylePart: image-based shape part manipulation

AbstractDirect part-level manipulation of man-made shapes in an image is desired given its simplicity. However, it is not intuitive given the existing manually created cuboid and cylinder controllers. To tackle this problem, we present StylePart, a framework that enables direct shape manipulation of an image by leveraging generative models of both images and 3D shapes. Our key contribution is a shape-consistent latent mapping function that connects the image generative latent space and the 3D man-made shape attribute latent space. Our method “forwardly maps” the image content to its corresponding 3D shape attributes, where the shape part can be easily manipulated. The attribute codes of the manipulated 3D shape are then “backwardly mapped” to the image latent code to obtain the final manipulated image. By using both forward and backward mapping, an user can edit the image directly without resorting to any 3D workflow. We demonstrate our approach through various manipulation tasks, including part replacement, part resizing, and shape orientation manipulation, and evaluate its effectiveness through extensive ablation studies.

The covariance environment defines cellular niches for spatial inference.

A key challenge of analyzing data from high-resolution spatial profiling technologies is to suitably represent the features of cellular neighborhoods or niches. Here we introduce the covariance environment (COVET), a representation that leverages the gene-gene covariate structure across cells in the niche to capture the multivariate nature of cellular interactions within it. We define a principled optimal transport-based distance metric between COVET niches that scales to millions of cells. Using COVET to encode spatial context, we developed environmental variational inference (ENVI), a conditional variational autoencoder that jointly embeds spatial and single-cell RNA sequencing data into a latent space. ENVI includes two decoders: one to impute gene expression across the spatial modality and a second to project spatial information onto single-cell data. ENVI can confer spatial context to genomics data from single dissociated cells and outperforms alternatives for imputing gene expression on diverse spatial datasets.

Deep conditional generative model for longitudinal single-slice abdominal computed tomography harmonization.

Two-dimensional single-slice abdominal computed tomography (CT) provides a detailed tissue map with high resolution allowing quantitative characterization of relationships between health conditions and aging. However, longitudinal analysis of body composition changes using these scans is difficult due to positional variation between slices acquired in different years, which leads to different organs/tissues being captured. To address this issue, we propose C-SliceGen, which takes an arbitrary axial slice in the abdominal region as a condition and generates a pre-defined vertebral level slice by estimating structural changes in the latent space. Our experiments on 2608 volumetric CT data from two in-house datasets and 50 subjects from the 2015 Multi-Atlas Abdomen Labeling Challenge Beyond the Cranial Vault (BTCV) dataset demonstrate that our model can generate high-quality images that are realistic and similar. We further evaluate our method's capability to harmonize longitudinal positional variation on 1033 subjects from the Baltimore longitudinal study of aging dataset, which contains longitudinal single abdominal slices, and confirmed that our method can harmonize the slice positional variance in terms of visceral fat area. This approach provides a promising direction for mapping slices from different vertebral levels to a target slice and reducing positional variance for single-slice longitudinal analysis. The source code is available at: https://github.com/MASILab/C-SliceGen.

Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification.

A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model's ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model's latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.

Tensor-based multi-view spectral clustering via shared latent space

Multi-view Spectral Clustering (MvSC) partitions data into clusters according to multiple views for higher performance. However, most existing works overlook model interpretability and involve iterative and alternating updates on parameters with expensive computations, where out-of-sample predictions are also commonly prohibitive. In this paper, we construct a novel weighted conjugate feature duality for formulating the MvSC problem cast as a weighted principal components analysis. In this new method, a common latent space is derived with shared dual variables, which achieves the couplings by learning projections from different views into the latent space. To boost higher-order correlations, the tensor-based modelling is introduced. Our method leads to a closed-form solution in the dual through a single eigendecomposition thanks to the primal–dual representation, leading to easy algorithmic setups in practice. Our novel conjugate duality and the conceived shared latent space greatly boost the efficiency in model representation and computation: element-wise operations of kernels are involved in the dual instead of the explicit tensor computation in the primal and the size of our eigendecomposition is independent of the number of views. Besides, we can effectively incorporate the visualization of the clusters in a single latent space, which is particularly useful with many data views. For predicting unseen data under generic applications, our method can be flexibly applied with out-of-sample extensions, which can also be utilized for further efficiency in large-scale cases with fixed-size kernel schemes. Numerical experiments verify our superiority in accuracy, efficiency, and diverse data explorations.